Weakly Coupled Deep Q-Networks

We propose weakly coupled deep Q-networks (WCDQN), a novel deep reinforcement learning algorithm that enhances performance in a class of structured problems called weakly coupled Markov decision processes (WCMDP). WCMDPs consist of multiple independent subproblems connected by an action space constraint, which is a structural property that frequently emerges in practice. Despite this appealing structure, WCMDPs quickly become intractable as the number of subproblems grows. WCDQN employs a single network to train multiple DQN "subagents", one for each subproblem, and then combine their solutions to establish an upper bound on the optimal action value. This guides the main DQN agent towards optimality. We show that the tabular version, weakly coupled Q-learning (WCQL), converges almost surely to the optimal action value. Numerical experiments show faster convergence compared to DQN and related techniques in settings with as many as 10 subproblems, $3^{10}$ total actions, and a continuous state space.Comment: To appear in proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023

Exploiting Structure and Relaxations in Reinforcement Learning and Stochastic Optimal Control

Stochastic optimal control studies the problem of sequential decision-making under uncertainty. Dynamic programming (DP) offers a principled approach to solving stochastic optimal control problems. A major drawback of DP methods, however, is that they become quickly intractable in large-scale problems. In this thesis, we show how structural results and various relaxation techniques can be used to obtain good approximations and accelerate learning. First, we propose a new provably convergent variant of Q-learning that leverages upper and lower bounds derived using information relaxation techniques to improve performance in the tabular setting. Second, we study weakly coupled DPs which are a broad class of stochastic sequential decision problems comprised of multiple subproblems coupled by some linking constraints but are otherwise independent. We propose another Q-learning based algorithm that makes use of Lagrangian relaxation to generate upper bounds and improve performance. We also extend our algorithm to the function approximation case using Deep Q-Networks. Finally, we study the problem of spatial dynamic Pricing for a fixed number of shared resources that circulate in a network. For the general network, we show that the optimal value function is concave and for a network composed of two locations, we show that the optimal policy enjoys certain monotonicity and bounded sensitivity properties. We use these results to propose a novel heuristic algorithm which we compare against several baselines

Photochemical and DNA degradation studies on tenoxicam, lornoxicam, and their photolysis products

Elghamry I, El-Ayaan U, Youssef MM, Al-Shihry S, Letzel M, Mattay J. Photochemical and DNA degradation studies on tenoxicam, lornoxicam, and their photolysis products. Monatshefte für Chemie. 2017;148(2):257-262.Tenoxicam and lornoxicam are nonsteroidal anti-inflammatory drugs, were subjected to photoirradiation at 254 nm led to the photodegradation of the pharmaceutical agents. Both, the isolated photodegradation products and the pharmaceutical agents were examined toward DNA binding and degradation. The photodegradation products degrade calf thymus in concentration dependent manner